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Human Interface System 

BACKGROUND 

The following description relates to human interface and input systems for electronic 
devices, particularly hand-held electronic devices, such as cell phones, personal digital assistants 
("PDAs"), pocket personal computers, smart phones, hand-held game devices, bar-code readers, 
5 remote controls, and other similar input devices having a keypad or one or more input elements. 

Electronic devices have become increasingly sophisticated and physically smaller due in 
part to a decrease in the price of processing power and a concurrent increase in demand by 
consumers for smaller devices. Such devices, however, tend to be limited in function and utihty 
by the user's ability to interface with the device for data input (e.g., text, numeric, and functional 
10 input) and/or device control, which becomes increasingly more difficult to do as the available 
space on the device's surface for positioning the input elements, which are used for data input 
and/or device control, continues to decrease. 

Various human interface and input systems and techniques for hand-held electronic 
devices have been developed for data input and device control. These include miniature 
15 keyboards and keypads used in combination with chordal input techniques, modal input 

techniques and/or smart keys; and touch screens used in combination with on-screen keyboard or 
keypad software or hand- writing recognition software. 

Keyboard or Key pad Used With Chordal, Modal and Smart Key Techniques 

Miniature keyboards and keypads are similar to their standard full-size versions — ^i.e., a 
20 keyboard generally has a full set or substantially full set of numeric, character, and functional 
input elements, while key pads typically have a reduced set of numeric, character and/or 
functional input elements compared to keyboards. These miniature input devices typically are 
designed to fit the available space on one surface of a hand-held electronic device or are 
designed as small, easily transportable, extemal plug-in devices. Thus, as hand-held electronic 
25 devices become smaller, the size of the input elements typically has been reduced in order for the 
desired number of input elements to fit on one surface of the electronic device. 

For data input and device control, miniature keyboards and keypads typically either 
require one of two input techniques- use of one or more thumbs or fingers to press the desired 
input elements or use of a stylus to "peck" the desired input elements (which is usually done 
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where the input element is of smaller size). Various techniques, such as chordal input 
techniques, modal input techniques and smart keys, have been developed and implemented to 
improve the efficiency and effectiveness of using miniature keyboards and keypads. 

• Chordal Input Techniques 

5 Chordal input techniques generally are based upon the principle that characters, symbols, 

words, phrases or concepts can be represented by a reduced set of input elements. Thus, by only 
having to press a reduced combination of input elements, functionality can be increased and 
quicker and more accurate data input can be realized. Chordal input techniques can be used on 
any keyboard or keypad configuration or any device having more than one input element, and 

10 typically resuhs in fewer input elements or more functions compared to conventional keyboards 
or keypads. An example of an electronic device using two-handed chordal input techniques is a 
court reporter or stenographer's typewriter. One chordal input technique using a keypad to 
decrease the number of actuations to achieve a large number of functions is described in U.S. 
Patent No. 5,973,621 to Levy, entitled "Compact Keyed Input Device," which is incorporated 

1 5 herein by reference. 

• Modal Input Techniques 

Modal input techniques are based on the concept that functions of the electronic device, 
e.g., text messaging in a cell-phone or PDA, can be accessed by pressing a particular input 
element (or combination of elements) on a keyboard or keypad. Once that particular input 

20 element is pressed, the functionality of all or a portion of the input elements on the keyboard or 
keypad may change. Modal techniques typically are used in calculators, cell-phones, and PDAs. 
For example, in cell phones, a modal technique called multi-tap is common, in which individual 
input elements on the keypad are associated with multiple symbols, such as characters, letters, 
numbers, icons or other types of symbols, which tends to reduce the number of input elements 

25 required to achieve the desired functions, e.g., a twelve-input-element keypad can be used to 
represent all letters of the Enghsh alphabet and the decimal digits. A user can input a desired 
symbol within a set of symbols associated with a certain input element by tapping on that 
particular input element with a thumb, finger, or stylus, one or more times to input the desired 
character. Thus, if a user desires to send a text message, the user may press a functional input 

30 element, e.g., a mode key, to access the text messaging functionality of the cell phone and then 
tap an individual input element one or more times to select the associated symbol for input. The 
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number of taps needed to input a particular symbol may differ depending on the language 
character set chosen. For example, Japanese keypad or keyboards typically require a minimum 
set of 46 characters for text input, while English or American keyboards and keypads usually 
require a minimum set of 26 characters for text input. These modal input techniques have gained 
5 some popularity as users perform more text functions. 
• Smart Keys 

Smart keys are typically used on keypads and refer to a single key or combination of keys 
that, when pressed, predict the users next logical action. Some implementations work better than 
others and some applications reduce the number of keystrokes required to complete a function 

10 better than others. Word-predictor software, for example, attempts to predict the word or 

character the user intends to input based upon one or more letters inputted by the user and the 
likely probabilities within a given language. The probability of the software guessing correctly 
increases with the length of the word or number of letters or characters inputted. In a device 
using smart keys on the keypad, a user may tap the keys 2, 2 and 8 in sequence to generate the 

15 word "cat" and the device would display that word first because it is usually the most conunon 
combination, whereas the word "bat," which can be generated by pressing the same keys, would 
not be displayed first because it is not as common. Also, the word "cat" may be displayed after 
pressing the 2 key the second time based on a guess by the word-predictor software. 

Smart keys also are typically used for Japanese data input where a user phonetically 

20 inputs letters representing the sound of the Japanese character (e.g., a Kanji character). Based on 
the inputted letters, the predictor software guesses the Japanese character. To select the 
character, a user would press the accept button or use the scrolling ftmction to go to the next 
character with a similar set of phonetic inputs. 

Touch Screen Using On-Screen Keyboard or Handwriting Recognition Software 

25 Using on-screen keyboard or keypad software with a touch screen offers users the ability 

to enter data with fingers or thumbs on a screen-sized keyboard or buttons, allowing faster data 
input without a stylus or physical keyboard or keypad accessory; while using handwriting 
recognition software with a touch screen, such as GrafFitiO on the Palm operating system, oflFers 
users the ability to enter text with a stylus by writing the text directly on the touch screen. Touch 

30 screens usually consume more power and are more expensive than non touch-sensitive screens. 
This higher power consumption can be a problem for hand-held electronic devices, which 



3 



16300-002001 / 

typically have limited power resources. Moreover, touch screens usually require the user to use 
both hands (e.g., one hand is used to hold and steady the device while the other hand is used to 
grasp the stylus), which is generally undesirable for interfacing with and controlling one handed 
hand-held electronic device, such as cell phones. Handwriting recognition software has 
5 improved the slowness and awkwardness inherent in stylus, finger or thumb input but other 
drawbacks still remain, such as high power consumption, the necessity to use both hands, and 
lack of tactile feedback to inform a user when an input element has been. Moreover, recognition 
software requires training to use properly, and, even then, still results in a high error rate. 
Game Control 

10 For game control, many of the above approaches have been used, but in most hand-held 

electronic devices, a user typically controls game play through the use of some form of input 
element, such as on a miniature keypad and/or directional pad ("D-pad"), which typically is 
located on the fi*ont surface of the device. Game control on some hand-held electronic devices, 
such as cell phones, is inherently one handed or at most two thumbed because of the size of the 

15 device, while game control on other hand-held electronic devices, such as PDAs and 

conventional game console controllers, is typically two-handed. The input elements associated 
with game control on these devices are typically digital even though analog input elements have 
been used on game controllers for PC and console game systems, such as Microsoft's Xbox or 
Sony's Play Station 2. 

20 SUMMARY 

The present inventors recognized that conventional human interface and input systems 
for hand-held electronic devices tended to be relatively inflexible, cumbersome, and inefficient 
to use, among other reasons, because they were not designed to take advantage of the 
biomechanics of the human hand, particularly the advantages associated with the opposition of 

25 the thumb to the fingers and the beneficial attributes of the thumb, e.g., its large range of motion 
and ability to impart large sustained forces, and the beneficial attributes of the fingers, e.g., their 
fine motor control, spatial memory and rapidity of motion. 

The present inventors also recognized that the input techniques developed to improve the 
efficiency of data input and device control, such as chordal and modal techniques, were limited 

30 by the inefficiencies inherent in conventional input systems. For example, miniature keyboards 
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and keypads used in combination with chordal input techniques not only required the user to 
memorize numerous input combinations and develop the necessary motor skills to control a 
reduced number of input elements to provide even more complex functionality compared to 
typical QWERTY keyboards, but also did not use or allocate input tasks to the fingers and thumb 
5 of the human hand effectively. Moreover, miniature keyboards and keypads used in combination 
with modal input techniques tended to limit the user's ability to efficiently input data depending 
on the number of taps required to input a particular symbol and how fast the user could tap the 
input element with his thumb or a stylus to select the particular symbol. 

The present inventors also recognized that a user's ability to control game play in such 

10 devices was greatly limited. For example, while analog game control has been available to users 
of PC and console game systems, analog game control generally has not been widely available 
on hand-held electronic devices, such as cell phones and PDAs. Moreover, because the game 
controls for conventional hand-held electronic devices were typically positioned on the front 
surface of the device, the user's hand typically obscured the user's view of the video screen. 

15 Also, the "fast twitch" control (e.g., a trigger) for shooting or activating special purpose game 
controls, which users have come to expect in console and PC game systems, generally has not 
been available to users of such hand-held electronic devices due in part to the nature of 
conventional interface and input systems, which were optimized for data input rather than for 
game control. 

20 Consequently, the present inventors developed a flexible and efficient human interface 

and input system and techniques for hand-held electronic devices (whether one handed or two 
handed) that utilize the opposed thumb and finger ergonomics inherent in the hand and the skills 
already developed for using conventional input techniques to accomplish data input, device 
control and game control in a timely, efficient, comfortable and intuitive manner. Thus, no 

25 specialized training beyond that normally expected with any newly purchased hand-held device 
is expected. 

Implementations of the human interface and input system for hand-held electronic 
devices described here may include various combinations of the following features. 

The human interface and input system for a hand-held electronic device may be 
30 configured to include on a first surface a plurality of input elements that can be manipulated by 
one or both of a human user's thumbs or a stylus. At least one of the input elements may be 
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configured in software to provide access to a plurality of input functions. For example, one of 
the input elements may provide access to the text symbols 5, j, k and 1, while another input 
element may provide access to the text symbols 3, d, e and f, such as is found on a typical cell 
phone keypad. The human interface and input system also includes on a second surface one or 
5 more selection elements that may be manipulated by any of the human user's fingers. The 

selection elements may be associated with one or more input functions, which may be configured 
in software. For example, the selection elements may be configured to correspond to a particular 
shift position. Therefore, when a user manipulates a selection element, which is configured to 
correspond to a third shift position, for example, then the input function that may be accessed by 

10 a particular input element will be the third input function associated with the input element. In 
the example provided above, the third input function may be the text symbol "k" for the input 
element that provides access to the text symbols 5, j, k and 1. 

One of the selection elements may be a pressure sensor pad that can be configured to 
represent a plurality of delineated active areas, as well as inactive areas. These delineated active 

15 areas likewise can be configured in software to represent one or more input functions. A shape 
changing media also may be provided with the pressure sensor pad so as to permit a hmnan user 
to tactilely discriminate between the plurality of delineated active areas and/or inactive areas. 

The input elements and/or selection elements also may be associated with a palpable 
detent, vibratory unit and/or force producing unit, which may provide tactile feedback to the user 

20 when the user manipulates the elements or in response to events occurring in a software 
application running on a processor. 

The human interface and input system also may be configured to include a first input 
assembly and a second input assembly. The first input assembly may include a plurality of input 
or selection elements situated on one or more surfaces of the electronic device and configured to 

25 be easily and comfortably actuated by one or both of a human user's thumbs or a stylus. The 
second input assembly may include one or more input or selection elements situated on one ore 
more surfaces of the electronic device and configured to be easily and comfortably actuated by 
one or more of the human user's fingers. The first input and second input assembUes may be 
disposed on one or more surfaces of the hand-held electronic device to take advantage of the full 

30 range of opposition configurations of the thumb and the fingers. Sensing circuitry, such as an 
input controller, may be provided to receive signals generated by the elements of the first and/or 
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second input assemblies when the elements are manipulated by the human user and convert those 
signals in a form suitable to be received by a processor running application software, which 
based on the received signals, can determine the type of input provided by the human user. 

The first input assembly may be situated on a front surface of the electronic device, while 
5 the second input assembly may be situated on the back surface of the electronic device to take 
advantage of the thumb/finger opposition. As configured, a user may manipulate input elements 
in the first input assembly with one or both thumbs or a stylus, while, manipulating elements in 
the second input assembly with one or more fingers. 

The input fimction of the input elements of the first and/or the second input assembly 

10 may change depending on the application running on the electronic device. When a text 

application (e.g., e-mail, word processing, or text messaging) is running on the electronic device, 
the elements of the first and/or second input assembly may be associated with data input keys, 
such as symbols. When a game application is running on the electronic device, the input 
elements of the first and/or second input assembly may be associated with game controls, such as 

15 a directional control, action buttons, and trigger buttons. The mapping of one ore more of the 
input elements of the first and/or second input assembly to a software application, i.e., whether 
one ore more of the input elements will operate as data input keys, game controls or device 
controls can be customized by the software application developer or the user through downloads 
or other programming modalities. Moreover, to reduce the cost of manufacturing hand-held 

20 electronic devices that will be used in multiple countries, input element sets particular to the 
language of a desired country can be implemented in software. 

The systems and techniques described here may provide one or more of the following 
advantages. The human interface and input system and associated techniques offer the 
fimctionality of a high performance game controller, which can support the most demanding 

25 game input requirements, and the speed and accuracy of data input that can be obtained with the 
use of a conventional standard QWERTY keyboard, but witiiout the large footprint. Also, the 
human interface and input system and associated techniques can increase the number of 
fimctions that may be associated with a given number of input elements without increasing the 
number of keystrokes or taps that is required. Moreover, it allows the input element size to 

30 remain consistent with the ergonomics of the human hand without increasing the time it takes to 
leam how to use the input system compared to conventional input systems. 
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Details of one or more implementations are set forth in the accompanying drawings and 
the description below. Other features and advantages will be apparent from the description and 
drawings, and from the claims. 

DESCRIPTION OF DRAWINGS 

Fig. 1 is a block diagram of a typical hand-held electronic device upon which the human 
interface and input system may be implemented. 

Fig. 2 is a block diagram of an implementation of the human interface and input system. 

Figs. 3a and 3b show front and back isometric views, respectively, of a hand-held 
electronic device wherein the second input assembly includes a pressure sensor pad having a 
plurality of configurable active areas. 

Fig. 3c illustrates an exploded view of an example of an input element of the first input 
assembly. 

Fig. 3d depicts one implementation of how the plurality of configurable active areas of 
the pressure sensor pad of the second input assembly may be configured. 

Figs. 4a and 4b depict front and back isometric views, respectively, of a hand-held 
electronic device wherein the second input assembly includes three touch pads. 

Figs. 5 a and 5b depict front and back isometric views, respectively, of a hand-held 
electronic device wherein the second input assembly includes three two-position rockers. 

Figs. 6a and 6b illustrate front and back isometric views, respectively, of a hand-held 
electronic device wherein the second input assembly includes a D-pad and two contact sensors. 

Figs. 7a and 7b show a two-handed hand-held electronic device wherein the second input 
assembly includes two rotary dials. 

Fig. 8 is a block diagram of a hand-held electronic device in the context of a 
communication system that may be used to implement the human interface and input systems 
and techniques described here. 

Like reference symbols in the various drawings indicate like elements. 

DETAILED DESCRIPTION 
Biomechanics of the Human Hand 

The human hand comprises an opposable thumb and four fmgers, i.e., the thumb may be 

used in opposition, in concert, in combination or ftmctional relation with any of the foxu- fmgers. 
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Compared to the human fingers, the hxmian thumb may be characterized as having larger range 
of motion, stronger sustaining force actuation and poorer dexterity. The human base joint of the 
thumb has three degrees of freedom, such as side-to-side movement, up and down movement, 
and rotation about the thumb's long axis; while the base joint of the fingers has two degrees of 
5 fireedom, such as side-to-side and up and down movement. Thus, the thumb typically is 

considered to have better range of motion than any of the fingers. Also, because the human 
thxmib has a bigger actuation muscle than any of the fingers, it can provide larger sustaining 
forces than the fingers. But also because of the larger muscle, the human thumb may suffer fi"om 
diminished fine motor control and rapidity of motion that can be exerted compared to the fingers. 
10 Thus, the human fingers are more suitable for performing tasks that require fine motor 
coordination or the ability to pinpoint or rapidly repeat actuation. 
Hand-Held Electronic Device Hardware Overview 

FIG 1 is a block diagram that illustrates a hand-held electronic device 100, such as a cell- 
phone, PDA, pocket PC, or smart phone, or other similar input devices upon which the human 

15 interface and input system and associated input techniques described herein may be 

implemented. Electronic devicelOO may include a bus 102 or other communication mechanism 
for communicating information, and a processor 104, such as an ARM, OMAP or other similar 
processor, coupled with bus 102 for processing information, such as one or more sequences of 
one or more instructions, which may be embedded software, firmware, or software appUcations, 

20 such as a text messaging application or video game. Electronic device 100 also may include a 
main memory 106, such as a random access memory (RAM) or other dynamic storage device, 
coupled to bus 102 for storing information and instructions to be executed by processor 104. 
Main memory 106 also may be used for storing temporary variables or other intermediate 
information during execution of instructions to be executed by processor 104. Electronic device 

25 100 further may include a read only memory (ROM) 108 or other static storage device coupled 
to bus 102 for storing static information and instructions for processor 104. A storage device 110 
may be provided and coupled to bus 102 for storing information and instructions. Electronic 
device 100 may also include a display 112, such as a liquid crystal display (LCD), for displaying 
information to a user, and a human interface and input system 114 for commvmicating 

30 information and command selections to processor 104. Electronic device 100 also may include a 
communication interface 118 coupled to bus 102. Communication interface 118 provides a two- 

9 
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way data communication coupling to a base station. For example, communication interface 118 
may be a wireless link, a modem to provide a data communication connection to a corresponding 
type of telephone line or any other communication interface known to one of ordinary skill. As 
another example, communication interface 118 may be a local area network (LAN) card to 
5 provide a data communication connection to a compatible LAN. In the wireless link 

implementation, communication interface 118 may send and receive electrical, electromagnetic 
or optical signals that carry digital data streams representing various types of information. 
Human Interface and Input System Overview 

FIG 2 is a block diagram illustrating the major subsystems of the himian interface and 

10 input system 114. The input system 114 may include a first input assembly 206, a second input 
assembly 208, and an input controller 216. The first input assembly 206 and the second input 
assembly 208 may include one or more input or elements. The input or selection elements, 
which may be keys, buttons, pressure sensor pads, touch pads or other actuators, associated with 
one or more sensors, which produce one or more electrical signals 214 when the input or 

15 selection elements are actuated. The input controller 216, which may include one or more 

processors, receives the one or more electrical signals 214 and converts them into a form suitable 
to be received and interpreted by processor 104 after passing through bus 102. 

One or more context signals 222 are provided to input controller 216 through bus 102 in 
response to processor 104 executing embedded software, firmware, or software applications, 

20 such as a text messaging application. The context signals 222 are received and used by input 
controller 216 to map input or selection elements in the first input assembly 206 and/or the 
second input assembly 208 to one or more application input fimctions and responses. For 
example, if a text application is being executed by processor 104, then the input controller 216 
may map one or more input elements of the first input assembly 206 to one or more symbols, 

25 such as characters, letters, numbers, icons, other types of symbols, or combinations of different 
types of symbols, and map one or more input or selection elements of the second input assembly 
208 to a shifting or indexing fimctionality. If processor 104 is executing a game apphcation, then 
the input controller 216 may map the input or selection elements of the input assemblies 206, 
208 to game fimctions. The mapping of the input or selection elements to particular input 

30 fimctions for a given software application, whether done by the input controller 216 or processor 
104, may be customized by the apphcation developer or the user through downloads or other 
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programming modalities. Moreover, the mapping of the input or selection elements may be done 
for language key set changes, which may reduce the cost of manufacturing hand-held electronic 
devices for manufacturers servicing multiple coxmtries. 

Alternative implementations of the input system 114 need not have the input controller 
5 216, particularly where cost is of a concern. In those instances, processor 104 may assume the 
functions of the input controller 216. Thus, processor 104 can perform the mapping function 
described above. 

Human Interface and Input System and Techniques Implementations 

Figs. 3a and 3b illustrates front and back isometric views, respectively, of a hand-held 

10 electronic device 302 upon which an implementation of the human interface and input system 
may be implemented. Electronic device 302 may include six planar or contoured surfaces: a 
front-side surface 312, a back-side surface 314, a lefl-side surface 316, a right-side surface 318, a 
top-side surface 320, and a bottom-side surface 322. Although, in other implementations, 
electronic device 302 may have more or fewer planar and/or contoured surfaces. On the front- 

15 side surface 3 12, a display 330, such as an LCD, and a first input assembly 340 is disposed 

adjacent to each other. Altematively, display 330 may be on a separate assembly such as those 
displays for PDA's and cell phones with a swivel-mounted screen or flip-phone configurations. 
Also, the first input assembly 340 may be disposed on more than one surface. The first input 
assembly may be a typical cell-phone keypad, which may include twelve input elements 342, 

20 although any number of input elements may be provided. A user's thumb or thumbs or a stylus 
may actuate the input elements 342. 

A second input assembly 350 is disposed on the back-side surface 314, lefl-side surface 
316 and right side surface 318. Altematively, the second input assembly may be disposed on one 
of those surfaces or a combination of those surfaces. In this implementation, the first input 

25 assembly 340 is disposed relative to the second input assembly 350 to take advantage of the 

opposition of the human thumb and finger. The second input assembly 350 includes a pressure 
sensor pad 354 having a plurality of software configurable active areas, which may be actuated 
by one or more of the user's fingers. The pressure sensor pad 354 in this implementation may 
include an actuator, such as an elastomeric material, attached to a force sensitive resistor array, a 

30 capacitive mat or array, or other similar pressure sensitive device or grid that can provide 

multiple outputs corresponding to the pressure readings of the plurality of active areas on the 
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pad's 354 surface. Here, the pressure sensor pad 354 wraps around from the left-side surface 316 
across the back-side surface 3 14 to the right-side surface 318. 

It is to be understood that the input elements 342, 354 in this implementation and any 
other implementation could be analog and/or digital buttons, keys, rockers (which may be a one 
5 or more position buttons or an analog joystick-type button), shders, dials or touch pads used in 
combination with pressure sensors (such as force sensitive resistors, piezo resistive sensors, and 
capacitive sensors), positional sensors (such as rotary encoders, linear potentiometers and the 
like) or other sensors or a combination of them. 

Fig. 3c depicts an exploded view of an input element 342 of the first input assembly 340, 

10 which is mapped to represent one or more text fimctions. Here, the input element is mapped to 
represent the number 7 and letters p, q, r and s, as is found on a typical keypad of a cell phone. 
Other input elements 342 may be associated with other letters, numbers and/or icons. For 
example, one input element may be associated with the nimiber 4 and letters g, h and i, while 
another input element may be associated with the number 2 and the letters a, b and c. 

15 As shown in FIG 3d, the pressure sensor pad 354 may be configured in software to 

represent one or more delineated active areas corresponding to different programmable fimctions 
depending on the application. In this case, inverted U-shaped active area 360 forms an active 
area- the vertical sides 362 of the inverted U-shaped active area 360 are on the left-side surface 
316 and the right-side surface 318 and the horizontal side 364 of the inverted U-shaped active 

20 area 360 is along the top edge of the pressure sensor pad 354 on the back-side surface 314. 
Below the inverted U-shaped active area 360 on the back-side surface 314 are five oblong- 
shaped active areas 372 labeled from 1 to 5. On the bottom of both the left-side surface 316 and 
the right-side surface 318 and stretching to the back-side surface 314 of the pressure sensor pad 
354 are rectangular-shaped active areas 374, 376, 378, 380. The remaining area of the pressure 

25 sensor pad 354 may be configured to be inactive. 

In this implementation, inverted U-shaped active area 360 may be used for navigation- 
the vertical sides 362 for y-directional movement and the horizontal side 364 for x-directional 
movement. The oblong-shaped active areas 372 may be used for shifting or indexing between 
symbols, such as characters, letters and/or numbers, or text input. The rectangular-shaped active 

30 areas 374, 376, 378, 380 may be used for shifting between modes — two of the active areas 374, 
376 for left-handed users and the other two active Ereas 378, 380 for right-handed users. In 
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another configuration of the pressure sensor pad 354, the entire surface of the pressure sensor 
pad 354 may be covered by horizontal rectangular active areas interspersed between small 
rectangular horizontal inactive areas to achieve any desired number of active areas. Other 
configurations of the pressure sensor pad 354 may be realized depending on the requirements of 
5 the desired application. 

The delineated active areas of the pressure sensor pad 354 may be actuated by one or 
more of the user's fingers, such as by applying pressure against the delineated active areas of the 
pad 354 beyond a pre-defined or user-adjustable threshold pressure. Likewise, the absence of 
pressure may be used as an actuation event. The pressure sensor pad 354 also may contain or be 

10 mounted above or below a shape-changing media such as an electrorheastatic fluid, shape 
memory metal array or similar material, which can permit the user to tactilely discriminate 
between the one or more delineated active areas. Thus, the user will be able to perceive the one 
or more delineated active areas as if they were physical buttons. Also, a computer graphical 
representation (not shown) of the configuration of the delineated active areas of the pad 354 may 

15 be displayed temporarily (or some predetermined time) on a portion of the display 330 to 
visually assist the user in locating where the delineated active areas of the pad 354 are 
positioned. Moreover, an input element 342 of the first input assembly 340 may be mapped to 
activate and/or deactivate the displaying of the computer graphical representation. 

The input architecture described above, with the first input assembly 340 on the front- 

20 side surface 312 and the second input assembly 350 on the back-side surface 314, left-side 

surface 316 and right-side surface 318, is configured to take advantage of the biomechanics of 
the hand, whether the user is left-handed or right-handed. This configuration, for example, can 
reduce the nimiber of thumb taps required to input a particular symbol compared to the number 
of thumb taps or stylus presses required using only a typical key pad with modal input 

25 techniques, such as is found in conventional text input systems. Moreover, this configuration can 
permit fiiU keyboard capability with fewer input elements on the first input assembly 340 and 
with greater spacing between input elements to enhance the ease of input compared to typical 
keypads for existing cell phones. Also, this configuration can permit fiiU fimctionality of a high 
performance game controller, which can support the most demanding game input requirements. 

30 A method to implement fixU keyboard capability and reduce the number of thumb taps is 

to map in software the delineated active areas of the pressure sensor pad 354, such as the oblong- 
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shaped active areas 372, to an indexing or shifting functionality to take advantage of the 
capability of the human finger, i.e., rapidity of motion, spatial memory and fine motor control, 
and to map in software the input elements 342 of the first input assembly 340 to text 
functionality to take advantage of the capability of the human thumb, i.e., range of motion and 
5 sustained force actuation. 

When a text messaging application is running on the electronic device 302 and displayed 
on the screen 330, the first input assembly 340 and the second input assembly 350 are used 
together to perform the text messaging functions. Each input element 342 of the first input 
assembly 340 may represent one or more text functions, e.g., one input element may be 
10 associated with the decimal digit 2 and letters a, b and c, while another input element may be 

associated with the decimal digit 7 and letters p, q, r and s (as shown in Fig. 3c), such as is found 
on typical keypads. 

In this implementation, the input elements 342 are configured the same as a typical 
keypad on a cell phone. The specific text function inputted by a user for a particular input 

15 element 342 is determined by which delineated active area of the pressure sensor pad 354 is 
pressed. For example, going from left to right, each oblong-shaped active area 372 may be 
mapped to represent a separate index or shift position such that index position 1 may be assigned 
to the left-most oblong-shaped active area (labeled 1), index position 2 may be assigned to the 
adjacent oblong-shaped active area 372 (labeled 2) and so on, where index position 5 may be 

20 assigned to the right-most oblong-shaped active area 372 (labeled 5). Thus, to input the word 
"yes", the user may press the oblong-shaped active area 372 representing index position 4 with 
any of his fingers and press the particular input element 342 representing the letter "y" with his 
thumb; then the user may press the oblong-shaped active area 372 representing index position 3 
with any of his fingers and press the input element 342 representing the letter "e" with his 

25 thumb; and then the user may press the oblong-shaped active area 372 representing index 

position 5 with any of his fingers and press the input element 342 representing the letter "s" with 
his thumb. 

The coordination of finger motions and thumb motions in other than a grasping motion 
may be difficuU for most people. Generally, doing two separate types of motions simultaneously 
30 can be difficult. However, the human interface and input system described herein does not 

require those types of motions due to the flexibility of the system. Generally, it is easier to tap 
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both the fingers and thumbs or leave either the thumb or fingers in contact with an input element 
or delineated active area while moving the other. For example, a user's finger may press an 
oblong-shaped active area 372 at the same time or nearly the same time the user's thumb taps an 
input element 342 in the first input assembly 340. 
5 Also, a user may tap an input element 342 with his thumb while pressing an oblong- 

shaped active area 372. Pressing or touching an oblong-shaped active area 372 while tapping on 
an input element 342 in the first input assembly 340 typically is natural, comfortable and easy to 
do. Likewise, the same holds true where the index finger moves substantially linearly fi-om one 
oblong-shaped active area 372 to the next, generally a left to right motion or vice versa, while the 

10 thumb taps an input element 342 in the first input assembly 340. 

Another way to implement finger/thumb coordination would be to permit asynchronous 
or sequential tapping between the finger tap and the thumb tap. For example, pressing an input 
element 342 within a pre-determined time (e.g., one second) after pressing and depressing a 
oblong-shaped active area 372 would constitute the same action as if both were pressed 

15 simultaneously. This time window could be configured by the user to facilitate different 
proficiencies in typing or different types of applications — for game applications, the time 
window could be quite short, whereas for text input applications, the time window could be 
much longer. The time window also could be different for different delineated active areas based 
on their intended fimction in a given application. 

20 Another method to implement ftiU keyboard capability and reduce the number of thumb 

taps is map in software the delineated active areas of the second input assembly 350 as follows: 
left vertical side 362 of the inverted U-shaped active area 360 to be shift position 1; anywhere 
along the horizontal side 364 to be shift position 2; the top-left rectangular-shaped active area 
378 to be shift position 3; the top-right rectangular-shaped active area 374 to be shift position 4; 

25 the bottom-left rectangular-shaped active area 380 to be shift position 5; and, if needed, the 
bottom-right rectangular-shaped active area 376. The input elements 342 of the first input 
assembly 340 may again be mapped to text fimctionality. 

Thus, to input the word "yes", the user may press the top-right rectangular-shaped active 
area 374 representing shift position 4 with any of his fingers and press the particular input 

30 element 342 representing the letter "y" with his thumb; then the user may press the top-left 
rectangular-shaped active area 378 representing index position 3 with any of his fingers and 
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press the input element 342 representing the letter "e" with his thumb; and then the user may 
press the bottom-left rectangular-shaped active area 380 representing index position 5 with any 
of his fingers and press the input element 342 representing the letter "s" with his thumb. 

A method of implementing the functionality of a game controller is to assign in software 
5 the input elements 342 of the first input assembly 340 specific game Amotions to take advantage 
of the capability of the human thumb, i.e., range of motion and sustained force actuation, and to 
map in software the delineated active areas of the pressure sensor pad 354 of the second input 
assembly 350 analog control to take advantage of the capability of the human finger, i.e., rapidity 
of motion, spatial memory and fine motor control. Thus, as a user's index finger or middle 

10 finger slides fi-om left to right across the oblong-shaped active areas 372, the horizontal side 364 
of the inverted U-shaped active area 360, and/or the rectangular active area 370, the input 
controller (not shown) may interpret the motion as "increasing" a parameter such as speed, size, 
position, etc. Alternatively, the input controller may be progranmied to interpret different levels 
of applied pressure to these delineated active areas as the "increasing" parameter, i.e., increased 

15 pressured may represent increased speed, size, position, etc. 

In this implementation, the vertical side 362 of the inverted U-shaped active area 360 
may be programmed to represent the y-directional (vertical) movement of control of a character 
in a game, while the horizontal side 364 of the U-shaped active area 360 may be programmed to 
represent the x-directional (horizontal) movement. Movement into or out of the field of view 

20 may be controlled by the left and right rectangular buttons 374, 376, 378, 380, thereby allowing 
3-D control. Rapid firing of weapons may be accomplished by using the input elements 342 of 
the first input assembly 340 or one of the five oblong-shaped active areas 372, with each one 
representing a different weapon or action. Complex moves or mode shifts could be 
accomplished by combining input elements 342 of the first input assembly 340 with any 

25 delineated active area of the second input assembly 350. In this way, a game developer may 

optimize the mapping of delineated active areas based on the best configuration for a particular 
game. For example, a game developer could set up control configurations for novice users 
differently than for advanced users, such as mapping different numbers or sizes of delineated 
active areas, in order to reduce the learning time to be proficient and make game control easier 

30 for first time players. 
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Figs. 4a and 4b illustrate front and back isometric views, respectively, of a hand-held 
electronic device 402 similar to the device shown in Figs. 3a and 3b, except the second input 
assembly 450 includes three input or selection elements 454, 456, 458, which may be 
rectangular-shaped touch pads. Each touch pad 454, 456, 458 may transduce the location of the 
5 contact of an object or a user's finger anywhere on its surface. Also each touch pad 454, 456, 
458 may correspond to different programmable functions. Here, touch pad 454 may be disposed 
on the back-side surface 414; touch pad 456 may be disposed on the lefl-side surface 416; and 
touch pad 458 may be disposed on the right-side surface 418. 

In a hand-held device such as a cell-phone or PDA, the second input assembly 450 may 

10 include a touch-pad located diagonally on the back-side surface 414 with another touch-pad on 
the left-side surface 416 because a right-handed user's index finger typically is placed along a 
diagonal path on the back-side surface 414 wrapping around to the lefl-side surface 416. In that 
case, second input assembly 450 may include touch pad 454 and touch pad 456. A user's finger 
may finger may move along the length of the touch-pad strip 454 in order to select the desired 

15 function. For example, a far left portion of touch-pad 454 may be mapped to be index position 1 , 
a far right portion of touch-ad 454 may be mapped to be index position 5, and portions between 
the far-left portion and the far right portion of the touch-pad 454 may be mapped to intervening 
index positions. Alternatively, index position 1 may be mapped to touch pad 456 for right- 
handed users and mapped to touch pad 458 for left-handed users. Thus, in this implementation, 

20 text input is similar to that as described with respect to Figs. 3a and 3b. Other configurations of 
the active areas of the touch pads 454, 456, 458 are possible and can be tailored to specific 
applications. 

Figs. 5a and 5b illustrate front and back isometric views, respectively, of a hand-held 
electronic device 502 similar to the device shown in Figs, 3a and 3b, except the second input 
25 assembly 550 includes three input or selection elements 554, which may be actuated by any of 
the user's fingers, typically the user's index finger or middle finger or a combination of both. 
The input elements 554 in this implementation are conventional two-position rockers. Thus, the 
second input assembly 550 can provide six index positions at a relatively low cost with passive 
tactile feedback built in. 

30 Figs. 6a and 6b illustrate front and back isometric views, respectively, of a hand-held 

electronic device 602 similar to the device shown in Figs 3a and 3b, except the second input 

17 



16300-002001 / 

assembly 650 includes three input or selection elements 654, 656, 658. Input element 654 may 
be a D-pad input device and input elements 656, 658 may be either digital or analog contact 
sensors. The D-pad 654 may be mounted on the center of the back-side surface 614 and mapped 
in software to represent one or more index or shift positions. For example, the D-pad 654 may 
5 be mapped to represent four index positions with each compass heading of the D-pad (e.g., 

North, South, East and West) representing a different index position. A fifth index position could 
be mapped to orthogonal movement of the center of the D-pad 654 into the device 602. 
Altematively, the D-pad 654 may be mapped to represent eight index positions, e.g., the compass 
directions North, South, East, West, Northeast, Northwest, Southeast and Southwest may be 

10 mapped. The contact sensors 656, 658 may be used as mode fimctions, for firing weapons, or 
any other fimctionality specified by an application developer. 

Figs, 7a and 7b illustrate front and back isometric views, respectively, of a two-handed 
hand-held electronic device 702. A first input assembly 740 including a plurality of input 
elements 742 is disposed on the fi-ont-side surface 712. A second input assembly 750, including 

15 two input or selection elements 754, 756, is disposed on the back-side surface 714. In this 

implementation, the two input elements 754, 756 are rotary dials. Altematively, rotary dial 754 
may be disposed on the left-side surface 716 and rotary dial 756 may be disposed on the right- 
side surface 718. In a one-handed hand-held electronic device, such as a cell-phone, typically 
one rotary dial is needed if placed on the back-side surface 714 or two rotary dials are needed if 

20 placed on the left and right side surfaces 716, 718. Rotation of the rotary dials 754, 756 may be 
mapped in software to represent one or more index positions. The rotary dials 754, 756 may be 
implemented with detents so that the user can distinguish between separate index positions, i.e., 
tactile feedback may be provided to the user's finger(s). 

FIG 8 is a block diagram that illustrates a hand-held electronic device 800, such as a 

25 cell-phone or PDA, upon which the human interface and input system and associated techniques 
described herein may be implemented in a conununication system. Network link 820 typically 
provides data communication through one or more networks to other devices. For example, 
network link 820 may provide a connection through local network 822 to a host computer 824 or 
to data equipment operated by an Intemet Service Provider (ISP) 826. ISP 826 in tum provides 

30 data communication services through the world wide packet data communication network now 
commonly referred to as the "Intemet" 828. Network link 820 also could provide data 
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communication directly to the ISP 826 and Internet 828. Local network 822 and Internet 828 
both use electrical, electromagnetic or optical signals that carry digital data streams. The signals 
through the various networks and the signals on network link 820, which carry the digital data to 
and from electronic device 800, are exemplary forms of carrier waves transporting the 
5 information. 

Electronic device 800 can send messages and receive data, including program code, 
which includes one or more sequences of one or more instructions, through the network(s) and 
network link 820. In the Internet example, a server 830 might transmit a requested code for an 
application program through Intemet 828, ISP 826, local network 822 and network link 820. In 

10 one aspect, one such downloaded application may be for software games to be played on 
electronic device 800, which may obtain application code in the form of a carrier wave. 

In any of the above implementations, actiye and/or passive tactile feedback may be 
implemented. To provide passive tactile feedback, the input elements of the first and/or second 
input assemblies may be combined with a palpable detent, such as a dome cap or dome spring so 

15 that a user can tactilely perceive, through his fingers or thumbs, activation and/or deactivation of 
an input element. In one implementation, among others, the palpable detent may be positioned 
between the actuator and sensor components of the input elements. To provide active tactile 
feedback, one ore more vibratory units or force producing units may be mounted in the hand- 
held electronic device and activated to provide tap or index level or other information to a user. 

20 The vibratory unit may be an electric motor with an eccentric mass attached to the motor's shaft, 
a solenoid, a variable reluctance device, a loud speaker or any other vibrator that can provide 
tactile feedback. A force producing unit may be a solenoid in non- vibratory mode, a motor, non- 
vibratory actuators or any other actuator that can produce forces. A vibratory unit and/or force 
producing unit may be provided for each input element . In that case, the vibratory unit and/or 

25 force producing unit may be mounted below the input element so that when the vibratory imit 
and/or force producing unit is activated, the vibratory unit and/or force producing unit can push 
out the surface of the electronic device to a different level or position depending on the 
information to be communicated. Thus, in implementations using a pressure sensor pad or 
touch-pad as the input element, a stepped array may be configured to indicate higher and higher 

30 levels of index positions across the touch pad or pressure sensor pad. The vibratory units and/or 
force producing units may also be used to provide tactile feedback to indicate the momentary 
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achievement of an objective, such as target lock in game applications. Tactile feedback may also 
be accomplished by actuators, such as a solenoid, which changes the stiflBiess of the input 
element electronically or pushes against the user's hand or fingers to indicate an event of interest 
in the software application. 

The computational aspects described here can be implemented in analog or digital 
electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. 
Where appropriate, aspects of these systems and techniques can be implemented in a computer 
program product tangibly embodied in a machine-readable storage device for execution by a 
programmable processor; and method steps can be performed by a programmable processor 
executing a program of instructions to perform fimctions by operating on input data and 
generating output. 

The systems and techniques described above utilize the biomechanics of the thumb and 
fingers, i.e., it uses the fiinction of opposition, the fine motor control of the fingers, and the larger 
range of motion and stronger actuation provided by the thumb. By using the fingers and thumb 
in concert, the number of taps and time needed to accomplish a given fimction is reduced, the 
accuracy is improved, and the natural programming inherent in the human hand replaces the 
training required for other systems. 

A number of implementations have been described. Other implementations may include 
different or additional features. For example, other configurations of the one or more input 
elements of the first and second input assemblies may be realized. Also, the hand-held electronic 
devices described herein may have more or less than six planar or contoured surfaces. Moreover, 
the number of input elements in the first and second input assemblies are not limited to the 
number of input elements described in the implementations above. 

Also, the one or more input elements of the first and second input assemblies may be any 
input or selection type known to one of skill in the art, such as keys, buttons, touch pads, other 
types of pads, rockers, sliders, dials, contact sensors or other actuators associated with any 
sensor. Each sensor associated with an actuator may include digital momentary on/oflF switches 
or analog sensors, such as pressure sensors (e.g., force sensitive resistors, piezo fihn sensors, or 
capacitive sensors), or positional sensors (e.g., rotary or linear potentiometers or encoders), or 
other analog sensors known to those of ordinary skill, or accelerometers or gyroscopes. The first 
and second input assemblies may include a combination of these different types of input or 
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selection elements, which may be mounted in the configurations shown in the figures or 
imbedded within the device to permit control through motion of the overall device. 

Moreover, the methods to provide data input, device control or game control may be 
performed in a different order and still achieve desirable results. Accordingly, other 
5 implementations are within the scope of the following claims. 
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